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ABSTRACT 

This paper reports on an evaluation of the first 6 
months of operation for two new services developed by Bibliographic 
Retrieval Services (BRS): the School Practices Information Network ^ 
(SPIN)f a computer-supported network of educational information 
providers at all organizational levels, and the Schc^ol Practices 
Information File (SPIF), an online computer-searchable database of 
inform'atian of educational •programs, practices, and instructional 
materials. SPIN and SPIF are briefly described, the evaluation design 
is reviewed and the population surveyed is described, and the data 
sources and data collection methods for the evalua^tion survey are 
outlined. Findings are then presented for the following areas: the 
frequency of SPIN use, the utility of SPIN, problems and needed 
improvements in SPIN, frequency of u&e and utility of SPIF, and user 
awareness of SPIF. General conclusions and discussion of overall 
results close the "^report. Five data tables accompany- ^he text and the 
survey questionnaire is appended* (Author/JL) 
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Year One Evaluation of SPIN an* SPIF: 
Procedures and Findings ' 



Introduction 



This paper reports on the evaluation^ of the first six months of operation 
for the School Practices Information Network (SPIN) and the School Practices 

.Information Fil^ (SPIF), SPIN and SPIF are services developed by Bibliographic 
Retrieval Services, one of the world's largest suppliers of online computer 
search services,^, BRS is part of the Information Technology Group (ITG), a 

' world-wide organization engaged in all areas of information processing and 
communication, . 

SPIN is a computer-supported network of education information providers 
,at all organizational levels, particularly those that serve educators in 
local school districts. SPIF is an online, computer-searchable database 
pf information on educational programs, practices, and instructional materials. 
Both services are described comprehensively in other papers reported irf this 
* session, For-the purposes of the evaluation, it is important to reiterate 
key elements of the rationale for the creation of SPIN a(fd SPIF, These are: 



• Current nationwide sources of education information (e.g., ERIC, 
ECER) do not provide sufficient resources relating to programs, 
practices, and materials in a form that practitioners can access 
and use. An online searchable. file of such .information would^ 
provide a valuable resource for information providers and their 
clients , 

t There is a need for a nationwide communication and exchange 
' system for blinking information providing organizations at "all 
lev^s. Such* a system can promote the sharing of resources not' 
fpuni in existing publ ic^readily available files. 



c 



jn« in exi 
jrp^«^f 



The purp3W«^f the evaluation of SPIN/SPIF is to test the validity of 
the rationale - to determine the use and utility of the services. Subsequent 



t ■ 



'Sections pf this paper desclibe the phase one evaluation desiqn, evaluation 
procedares, and findings. ^ ^inal section provides concUsions, implications 
based on this fjrst round of evaluation activities, and a discussion of con- 
ceptual and operational issues rel-ated to the further development of this 
nationwide conl^tffrlcation, exchange and practice infortnation system. 

THe-^SPIF/SPIN system" was imtidted in late March; 1981.". Based on pre- 
yious experience with the pilot test f)f the National Education Practices 
File for the National Institute of Education, B4^S staff were a\^are that the 
growth of the Network and the use of the File would be gradual and sIqw. 
Given that summer^ is. a very low utilizatioa period for education information 
providers, it is safe to say that the SPI'N/SPIF system is in the verj^early 
stages of development. For this- reason the phase one evaluation, initiated' 
in Dece^nber, 1981, was viewed as a preliminary assessment. The focus was on 

■ If' 

the SPIN mefnbers and not on their clients: * - 



SPIN/SPIF Evaluation Design 

Evaluation Questions : - 

X 

The evaluation design for SPIN/SPIF. based an th^ following broad 

<^ ; . . , 

questions: 

1. How are JjfTfOrmati on -providers using. SPIN and SPIF?^ % 

2.. How can SPIN and SPIF be improved to increase their utility to 
information providers? • ^ 

' • i 

3. What current and futufe unmet inforifiation service need's should^ 
be addressed by SPIN and SPIF? 

A larger set of specific questions or variables of intere^ are vested 

within these broader information neecls. 'With respect to SPIN, program / 

managers had the following specif ic ,informat ion needs. " 
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t What is the frequency of use for each of the SPIN service's (1,e.i 
online member directory, onltne newsletter, and electronic 'mail )? 

t How do informatioit providers rate the utility of SPIN services-? 



t What kinds of comnunications ar^e transmi ttlfc-using^'electronic 

mail? . . 

t Does the SPIN system operate as designed? Can users operate 
the system? 



With respect to SPIF, program managers had the following information 
needs: 

• How frequently is the database searched? 

» 

• Is the search system operating designed? Can users search 
the database adequately? 

t Do information providers find the database useful ?^ Do tfjeir 
clients? Are some resources piore useful than others? If yes, 
which ones? 

t Is additional information spught from the contact person or 
organization cited in the abstract? How oftefi? Are satis- 
factory responses (i.e., more information or materials) pro- 
vided? . ^ 

t Is, the database royalty reasonable? Is th^e database judged 
more valuable than other education. databases generally 
available? ^ 

^ What other content is needed in SPIF? How else could SPIF 
be'^de more useful? 



Pata Sources 

The principal data sources for responding to these evaluation questions 
were the SPIN members. In addition, docu^ntation was maintained of key 
development and operational incidents. A SPIN Member Questionnaire was used 
to gather data relating to the specific evaluation questions (see Appendix A) 



Data Collection Methods, ' . 

The SPIN M^ber Questionnaire was seift to fifty-five SPIN members in - 
mid-Dece#iber, 1981. -Al though , the Netv/brk had nearly seventy-five riiembers 
at that point, prograw' tnanagers ^decided to survey only those who had been 
members throu^ October, '1981,. This was done tjO focus on members who had had 
sufficient time to become familjar with, and use, the system, A second* mail- 
ing was used for non-respondents. T^ephone intervfev/s were conducted to 
obtain further 'inf^rmation'when necessary, 

jiOf the fifty-five questionnaires sent out, forty-four (80 percent^ 
werfe returned. Table 1 displays' the return rate by type of organization. 
Of the forty-fdljr que'stionilairles returned, nine were not completed and were 
not included in the analysis, ^he reason given by all nine respondents was 
that' it was too soon for them to provide Evaluative judgments about the 
services; they had not had enough experience with it. Thus, a total of 
thirty-five questionnaires were u\Ld in the. analysis, ^ 

4 

4 

' ' ^ ' ' . ' V 

Evaluation Flndirigs . ' ' ^ 

Frequency of SPIN Use 

SPIN mernbers were asked to rate the frequency of their use of the three 
SPIN services *- the member directory, the newsletter, and electronic mail. 
Table 2' displays the mean responses by group for each service, A four-point 
scale was used by respondents, .with 1 indicating "never used", 2 indicating 
"rarely", 3 -"sometimes", and 4-"frequently, " Mean scores for all three * 
services ranged between rarely and sometimes, with the use of electronic 
mail slightly higher than the other two services. Respondents indicating 
no or rare use of"the services were asked to comment on their responses. 
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Tal}le 1 ' 
Return Rate by Organization Type 



Organization Type 


Number Sent 


Number (%) Returned Number Usable 


National 


6 


5 (83) ' 




4 


National-Regional 


12 


10 (83/ 




8 


State 


17 


• 

15 (88) 




13 


State-Regional 


10 


8 (80 ) 




c 
0 


School District^r' 


7 


, 5 (71) ■ 




4 


Publishers ^ 


2 - 


1 (50)^- 




0 


Higher Education 

• 


'J, 


0 -■ 




IT 


Total 55 


44 (80) 


• 


35 




Table 2 


















Frequency of Use of SPIN Serviced. 






Organization Type 


Di rectory ' 


Newsletter 


V 

Electronic 


Mail 


National (N=4) 


2.75 


2.5 ■ 


2.5 




National -Reg iOj>a1^(N=8) 


2.75 


. 2.5 


3.5 




State (N=13) 


2.92 


' 2.54 . - 


2.92 




State-Regional (N=6) 


2.5 . 


2.33 


2.33 




School Districts (N=4) 


•2.33 


' 2:75 


-2.5- 




X 


2.72 


2.51 


2.86 
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The most' frequent response was t^^^^ey were .not yet fully trained in the 
us^ of the system (five respotises). Three respondents indicated they hTad 
had no reason to use one or more of the services. Telephone interviews with 
four low-user respondents indicated that, although technically capable of 
using the services, such use had not become a part of the^r normal routi^ne. 
One respondent commented that she had no- use for the services. Four respon- 

r 

dents indicated that searching the online member directory was imnecessani ly 
time^onsupiing. 

SPIN Utility ^ ' 

Table 3 displays the mean ratings of utility of the three SPIN services. 

% 

A four-point scale was used by respondents, with 1 indicating "useless" and 
4 Indicating "very useful^" Mean scores varied from a high of 3.00 to a low 
of 2^25, indicating a relatively uniform response across groups. Mean 
responses were similarly uniform. Comments by SPIN members indicated that 
, non-respoTise was attributable to non-use or to insufficient use to support 
^' judgment. Other comments explaining low ratings (1 or 2) were: "need more 
training," "electronic mail needs more BRS promotion," and, "the online 
directory should be tn print form." 



Purposes for Using Electronic Mail ^ 
SP-IN members were a^ed for wh^^urposes they had used the electronic 



mail system. Table 4 displays theil^fesponses. Tjwo respondents indicated 
that they had never used the electronic mail system and three indicated that 
they had used the syst-em "once or twice, just to try it out." By far the 
most frequently indicated purpose was for. routine communication. Routine 



.1 


■ . Table 3 


• 






Utility of SPIN Services 




Organization Type 


Directory 

— ( ■ 


' * Newsletter 
* 


Electronic Mai 1 


National (N=4) 


3.00 (NR=1) 


2.71 (NR=1) 


2.50 . 


N;^ti nnal -Rpaional (N=8) 


2.71- (NR=2) 


2.71 (NR=1) 


2.43 (NR=1) 


i ' 


2.50 (NR=1) 


2.25 (NR=3) 


2.75 (NR=2) . ' 


State-Regional (N=6) 


" 2.33 


2.50 


2.50' ^ 


School Districts (t<=4) 


2.50 (NR=1) • 


, 2.75 (NR=1) 


2.50 (NR=«) 




2.56 


2.50 


2.58 . 


% 


♦ 




1 

\ 



Table 4 

Purposes for Using Electronic Mail 



Organization Type' 



Routine Receiving Responding to • Making 

Comnuni cation Requests for Requests for Reque«ts for 
Information Information Information 



National (N=4) ' 

National-Regional 
(N=8) 

State (N=13) 

State- Regional (N=6) 

School Districts 
(N=4)- 

*muliiple counts) 



7 

10 
4 



1 

4 

3 

1 
1 



5 
6 

2 



3 
7 
2 

1 
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comnunicatidn included such purposes as notifying another SPfN member o'f an 
upcoming event and passing along other unsolicited work-reTated information. 
A good portion of. routine communications appear to have been used fpf social^ 
exchange, unrelated directly to work. / , ^'r 

Considerably less communication was done for requesting or responding to' 
requests for information, although it appears that members rfi-d respond to 
requests for information more ;frequently than they received them. This 
discrepancy was'explained by two members who indicated that they used'the^ 
system for responding to requests for information that they h^d receivedTvia 
other channels (i.e., .mail or telephone). » 

Problems^and Needed Improvements in SPIN 

Three members indicated-fhat they had problems with the SPIN cpmmunica- 
tions system, twenty-nine cited no problems, and three did not respond to 
the question. The problems cited were: "takes a while to get a response," 
"'have had problems signing on," -^'I sent in a newsletter article and^never 
heard," and "the [onl inej directory is too cumbersome to use." Even members 
who indicated that they had no problems were able to offer suggestions* ^or 
improvements to the SPIN communications and networking system. Those™ cited 
were: 

^. • provide printed member directory; update 'every six months / 

• the indexing system for the online directory is helpful; tell people 
about ^it 

• need print newsletter* 

• need tmre training » * • * 

• promote use of SPIN 

^ ^ • allow for remote entry of newsletter articles 



• increase membef ship in vocational ^reai 
^ • get more people in Network 
^ • allow* for longer messages without signing off and ^on the system 

Telephone conversations .with SPIN members revealed that many were just 
getting used to a new way of coffinuni eating and that the experimentation period 
was just phasing out. Most members indicated that they planned /o use the 
system^more frequently in the future. 
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Frequency of Use and Utility of SPIF . 
* 

SPIN members were asked to indicate ttie frequency of their use of the 

SPIF database and their ratings of its utility. Table 5 displays the nisponses 

», ♦ * 

and demonstrates the generally higher mean ratings' for SPIF than for SPIN. 

Across all organization types ttie frequency of use ratings 'ranged from 2.75 
t - * 

to 3.23, viith a mean of 2.99. Utility ratings y^ere somewhat higher, ranging » 
from 3.00 to 3.50, with a mean of 3.20. Oaly two respondents indicated no 
use of SPIF and gave as .a reason Tack of adequate, training to search the 
database* ^o respondent indicated that the database was "useless," although 
three respondents stated that the file was still relatively small in size or 
scope, thus limiting utility. SPIN m'embers were aksed whether their clients ' 
found jnformation from the SPIF database useful. Twenty-seven responded 
positively, with the remainder not respondiiig. Bas^ij on eomments from mem- 
bers, non-response* indicated that' they had no way of ^determining clients' 
use or evaluative Judgments. . ^ 

Only eleven of the thirty-five^SPIF^ users (33 percefht) indicated that ^ 
they attempted to contact people cited in the abstracts to obtain additional 
information* Of these, all indicated that they were satisfied with the 
resppnse. The larg6 number of members not attempting to collect additional 



Table 5- 

Frequent of Use* and Utility of SPJF 



Organ i zati orv Tjrpe Frequency of Use . • Utility 



'National / 3.00 3.00 

} 

National-Regional * - - 2.88^ 3.1? 

# 

*State- 3.2?- 3.31 

State-Regional 2.75 3.00 

School Districts 2.75 * 3.50 

* . 

^ X 2.99 * 3.20 



N information was dtvidec^Hnto two groups. Four indicated they didn't try, 
and ei^teen (55 percent) indicated they left such follovZ-up to their clients. 
Two members providcKl no explanation for their non-response. The most typical 
request for information was for sample materials or for additional program 
descriptions. In^some c^ses, the additional information was requested and 
received through ^ telephone contact." ' v 

The majority of respondents (21/35, *64 percent) indicated that they 
found the SPIF database -to be more val uable^ than others available to them. 

X 

Eight (24 percent) found it less useful and four (12 percent) said it was 

♦ A 

about equally valuable. Comments accompanying the response of lower utility 

^ dealt with the relatively small size and scope of the file and with the need 

it. 

.for greater coverage in certain areas. *Jour members Indicated that the 

\ . ' 

royalty on the database ($10 per hour; $.10 per offline print) ^was not ^ 

reasonable,' but all other users said it^was reasonable^ 
* > ^ - / . 
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Only two users indicated that they had probVems in searching SPIF,., In 
both cases, the need l&f training was cited/ One of these respondents * • 
indicated that th^y had problems in signing on to th'e syste^i. Nearly half' 
6f the users (15/33) made suggestions for improving SPIF, Th^ mast frequently 

cited comments were: * ' 

S 

t Haven't used it e-nough to evaluate fully ^(4 responses)' 
t SPIF recor?ls hard to "decode" 

• Go after resources in ,high priority areas (2'*resp^5,es) 

L 

• Provjde special training for iPIF (3 responses) 
* _ 

V t Need for more school-level practices ^ 
t Some I'ecords too long (2 ^responses) 

t Add paragraph headings to make records self-explanatory (e.g.T 
Resource Type for RT) 

t Advertise beyond the online community 

k Need to be able to s^e ERIC search strategy and execute on SPIF 
Suggestions for addi'tional content for SPIF were prov/ided by six 
respondents. All others indicated they could think of^o other content to 
be added. Suggestions included: "more focus on programs," "more voca- 
tional educ?&tion curricula," and "more on bilingual education practices." 
Feur respondents indicated that more records are needed in most content 
areas, 

SPIF Awareness 

\ substantial majority of users (^27/33) -had undertaken no special 
awareness^ activities to advertise SPIE and its availability to their clients. 
^Six member organizations did undertake in-house awareness. The reason most 
often given to explain -the low incidence of awareness activities was the 



\ 
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lack of ^PIF awareness materials. Several members suggested that BRS 
provide materials^to them for use in their mailings to clients, 'A few 
* members indicated that clients very seldom asked for a search of a particu-, 
lar database, leaving 'it up 'to the information provideu^ to decide Jrfhat 
sources were most appropriate. 

Conclusions 

Based' on the data collected through the questionnaire and through 
telephone interviews, it appears "fchat users of SPIN and SPIF found the 
systems technically adequate and useful in responding to their clients' 
information needs. The relatively large number of respondents reporting 
that it was too soon to evaluate the services, and the small number of non- 
respondents , indicate that all findings shouldbe viewed as preliminary. 

Ratings'of frequency of use of SPIN services indicate that usage was 
moderate during the start-up period, with lack of training in system use 
being the most frequeptly cited explanation. Usage rates for SPIF were 
slightly higher but still were lessened by insufficient familiarity with 
searching the database. Utility ratings for SPIN were moderate and were 
significantly higher for SPIF, State level organizations tended to be higher 
users of SPIN and SPIF,; but the distribution across all organization types 
are too narrow to demonstrate significant differences. 

Ratings of SPIN'S u^fulness during the start-up period were moderate; 
those for SPIF were substantially ht^her. Despite problems normally 
associated with initiation (i.e., lack of sufficient training, experimenta- 
tion, and limited size and scope), respondents indicated a generally high 
level of satf%faction with SPIN and SPIF. The database is rated as equal* 
to or better than other av^lable files by a substantial majority of users.. 



% 

Respondenjts ' comments indicate that phenomena related to technology change 
were operating, that mem^rsw^l^jus t getting a feel for the system and 
beginning -to discover howi it might Ibe used. Several respondents indicated » 
that BRS should- take^more oT^leaaersKi role Jn advertising the service 
and demonstrating its users and .utjflity to current and potential members. 

Discussion . ^ 

Based on the findings from this first evaluation phase, it appears that 
SPIN is g^wi'ng rapidly in number of members, but somewhat less quickly in 
terms of use of SPIN/SPIF services. The anxiety of system managers that 
the evaluation might have been undertaken too soon is in part justified 
by the substantial number of respondents who indicated no, little, or 
moderate use. Another indication of this development was that nineteen, 
respondents indicated that they would be willing to work wdth BRS in collect- 
ing client evaluation information in the future . It is too soon to conduct 
a survey of actual "end-users*' of ^IF information. (Tt^e design for such 
an evaluation is complete and will be implemented during phase two evaluation 
activities. 

A few operational changes in SPIN/SPIF operations were made even before 
formal evaluation information was available. Informal feedback indicated 
that searching the print directory was tim^ consuming and costly. BRS staff^ 
prepared a printed directory and distributed it with the winter newsletter. 
Plans were made to update and distribute the directory with each quarterly 
newsletter. The online directory was to be used for updates occuring during 
the quarter. A printed version of the online newsletter was created in much 
the same way, in order to provide an indepth printed version of the online 
file. 

13 



Two significant content changes in the SPIF database were the inclusions 
of descriptions of commercial materials and instructional - software for mj'cro- 
computers. Both modifications were motivated by SPIN members and the file 
has begun to expand in these directions. Acquisition efforts in these and 
other areas are to receive' i|^creased attention in the coming year. 

• Because SPIN and SPIF are part of a BRS-^supported R&[^ project, its 
long-term future will be decided by .the nature and 1 evel^of ^market dem^d. , 
In the near term, ^Ehe R&P effort'-^wi 1 1 be guided by the evaluation of its 
users according to a wide range of process amd outcome measures. Early 
indications are' that, the services are generally sound and useful and that 
demand for them is substantial. These indicants are gross measures at this 
stage, and further indej^th assessment at both the information provider and 
end user levels will be necessary to substantiate and. illuminate preliminary 
findinas. . - 
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SPIN Member Questionnaire 



tour T# 



Person Responding 



4m 










How frequently do you use: 


Never . 


Rare'ly 


Sometimes 


• Frequently 


a» SPIN member directory 


1 


2 

• 


3 


4 


b. SPIN newsletter 


' 1. 


2 


. 3 


4 


v^c. SPIN electronic mail 


1 


2 


3 


4 


d. SPIF database 


1 


2 


3 


4 












If you circled a 1 -Of 2 for any service 
a. — - 


, please explain: 


* 


b. 


• 








c. 










d. 


.#: 






r 




How useful is the: 


Useless 






Very Useful 


a. SPIN member directory 


l" 


2 • 


3 


4 


b. SPIN newsletter 


1 


? 


3 


4 


c, SPIN efectronic mail 


1 


2 


f 

•3 


•4 


• 

d. SPIF database 


1 


2 


3 


4 


If you circled a 1 or 2 for 


any service 


, please explain: 

L 




a. 










b. 


1 








c. 
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With respect to SPIF: yes ' nov 

- Did your clients find the information useful? ~ 

- Did you contact people cited in the abstracts 

for more information? . 

• If yes, were you satisfied with the response? ^ 

• If no, why not? ' 



- Is the T*oyalty reasonable? 

- Is the database more valuable than others 
available to you? 

- E)^lain "no" 



- Do you know how to submit items for ^ ^ 

inclusion in SPIF? ^ ^ 

Is there other content that you would like to have 

available through SPIF? ^ 

If yes, specify 



5. Have you undertaken any snecial awareness activities to advertise the 
availability of SPIF? 

I yes no 

Comment: 



6. Have you encountered any problems in searching SPIF? 



yes ; no 



Comnerit: 



1 r 
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What suggestions do you have for improving SPIF? 

a. Content of SPIF 

• 4 

b. Format of SPIF records 

c. Search procedures 

d. SPIF services 




For what purposes have you used electronic maiT (check all that apply) 
] never used 

^ routine communication 

' receiving requests ^or Information 

responding to requests for Information 
[ making requests 

other» 



9. Have you encountered any problems in using the SPIN connuni cations 
system? ' 

□ yes IZJno 



Comment: 



10. What suggestions do you have for improving SPIN? 



11. Do you evaluate your information services using client feedback? 
I lyes I I no 

If|^es, describe briefly. 



12, Would you, be willing to work with ESG in collecting evaluation 
^ information from your clients on the utility of SPIF? 



□yes . O 



no 



